Bayesian Sequential Designs

Jörn Alexander Quent

29 July 2020

What will I talk about?

What will I talk about?

Let’s talk about the frequentist way

\[t = \frac{\overline{x} - \mu}{s_\overline{x}}\] where \[ s_\overline{x} = \frac{s}{\sqrt{n}} \] where

\(\mu\) = Proposed constant for the population mean
\(\overline{x}\) = Sample mean
\(n\) = Sample size (i.e., number of observations)
\(s\) = Sample standard deviation
\(s_\overline{x}\) = Estimated standard error of the mean

Let’s talk about the frequentist way

Our toy example:

set.seed(911225)
n    <- 60
data <- rnorm(n, 0.0, 1)

df             <- n - 1
standard.error <- sd(data)/sqrt(n)
t.value        <- (mean(data) - 0)/standard.error 

Let’s talk about the frequentist way

A Bayesian way

\[\underbrace{\frac{p(H_1 | data)}{p(H_0 | data)}}_\text{Posterior plausibility about hypotheses} = \underbrace{\frac{p(H_1)}{p(H_0)}}_\text{Prior plausibility about hypotheses} \times \underbrace{\frac{p(data| H_1)}{p(data| H_0)}}_\text{Bayes factor = Predictive updating factor}\]

Evidence in favour of alternative:

\[ BF_{10} = \frac{p(data| H_1)}{p(data| H_0)}\]

Evidence in favour of null: \[ BF_{01} = \frac{p(data| H_0)}{p(data| H_1)}\]

This mean that you can just inverse them: \[ BF_{10} = 1/BF_{01}\]

Convention for Bayes Factors

\(BF_{10}\) Evidence
> 100 Extreme evidence for \(H_1\)
30 – 100 Very strong evidence for \(H_1\)
10 – 30 Strong evidence for \(H_1\)
3 – 10 Moderate evidence for \(H_1\)
1 – 3 Anecdotal evidence for \(H_1\)
1 No evidence
1 – 1/3 Anecdotal evidence for \(H_0\)
1/3 – 1/10 Moderate evidence for \(H_0\)
1/10 – 1/30 Strong evidence for \(H_0\)
1/30 – 1/100 Very strong evidence for \(H_0\)
< 1/100 Extreme evidence for \(H_0\)

A Bayesian way

library(BayesFactor)
ttestBF(data)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 3.236673 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS

A Bayesian way

\[s_\overline{x} = \frac{s}{\sqrt{n}} \;\;\;\;\;\;\;\; t = \frac{\overline{x} - \mu}{s_\overline{x}}\]

Rouder et al. (2009)

Traditional design example

This shows that a large sample size is necessary to a probability of 80% that \(BF_{10}\) > 10 or \(BF_{10}\) < 1/6.

Effect size Necessary sample size Misleading evidence in % £ for online experiment £ for fMRI experiment
0.5 72 0.0003 605 39600
0.0 232 0.0011 1949 127600

Sequential design

Sequential design

Sequential design



Effect size Average sample size Maximal sample size Misleading evidence in %
0.5 41 170 0.13
0.0 83 2765 2.95

Pure sequential design example


Costs for traditional designs:

Effect size Necessary sample size Misleading evidence in % £ for online experiment £ for fMRI experiment
0.5 72 0.0003 605 39600
0.0 232 0.0011 1949 127600


Costs for pure sequential design:

Effect size Average sample size Maximal sample size Misleading evidence in % £ for online experiment £ for fMRI experiment
0.5 41 170 0.13 344 22550
0.0 83 2765 2.95 697 45650

Setting an upper limit

Setting an upper limit



Effect size Average sample size Strong evidence in % Misleading evidence in % Insufficient evidence in %
0.5 39 98.05 0.12 1.83
0.0 58 80.25 2.31 17.44

Advantages of sequential designs

But

General remarks

Concluding words

The CBU should use sequential designs more often.

Further reading and material

All code can be found here. Further points and more simulations can be found here.